Skip to content

Move Feature Flagging evaluation into the provider runtime - #12097

Draft
leoromanovsky wants to merge 4 commits into
masterfrom
leo.romanovsky/feature-flagging-library-runtime
Draft

Move Feature Flagging evaluation into the provider runtime#12097
leoromanovsky wants to merge 4 commits into
masterfrom
leo.romanovsky/feature-flagging-library-runtime

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Motivation

Before this change, Java Feature Flags required the application to start with the Datadog Java agent:

java -javaagent:/path/to/dd-java-agent.jar -jar application.jar

The provider delegated UFC parsing and evaluation to classes owned by the Java agent. Without -javaagent, the provider could not load configuration or evaluate flags.

Despite its name, feature-flagging-lib was never a reusable Feature Flagging library. Since its introduction, it owned agent-side Remote Configuration and telemetry delivery. It depended on agent communication, bootstrap, configuration, and tracer classes. It was not published in dd-openfeature and could not run as a provider-only dependency.

Reusing that module would retain the agent coupling. This change creates a dependency-light provider core. It renames the historical module to describe its agent-runtime role.

Managed serverless platforms make the startup requirement costly:

  • AWS Lambda users must add the Datadog Java Lambda layer and set JAVA_TOOL_OPTIONS=-javaagent:/opt/java/lib/dd-java-agent.jar.
  • Google Cloud Functions users must package agent JARs and load them through JAVA_TOOL_OPTIONS. First-generation functions require two agents.
  • Azure Functions users must package two agents and configure languageWorkers__java__arguments or JAVA_OPTS, depending on the hosting plan.

These steps couple flag evaluation to tracing packages, platform-specific runtime configuration, and agent compatibility. The default CDN path must work with dd-openfeature alone.

The project must also preserve Remote Configuration. Existing customers need the agent-backed source. Released providers must continue to work with a new agent.

Changes

The default OpenFeature startup is now provider-only:

OpenFeatureAPI api = OpenFeatureAPI.getInstance();
api.setProvider(new Provider());
Client client = api.getClient("checkout");

The application supplies DD_API_KEY, DD_ENV, and DD_SITE. It does not set a configuration-source selector or attach -javaagent. The provider starts and owns the internal CDN poller.

ffe-dogfooding #102 uses this startup path. The script runs java -jar app.jar unless remote_config is explicit.

HTTP cancellation now handles both direct and ExecutionException-wrapped CancellationException results. Both paths return InterruptedIOException.

The published coordinate remains com.datadoghq:dd-openfeature. The provider JAR embeds feature-flagging-core and feature-flagging-http.

flowchart TD
    APP["Application and OpenFeature SDK"] --> API["feature-flagging-api<br/>published as dd-openfeature"]

    subgraph PROVIDER["Provider-owned modules embedded in dd-openfeature"]
        API --> CORE["feature-flagging-core<br/>UFC model, parser, evaluator,<br/>snapshot, last-known-good state"]
        API --> HTTP["feature-flagging-http<br/>Java 11 HttpClient, ETag,<br/>retry, scheduling, lifecycle"]
        HTTP --> CORE
    end

    API -. "reflective JDK-only calls" .-> BRIDGE["feature-flagging-bootstrap<br/>raw compatibility bridge"]

    subgraph AGENT["Java agent modules"]
        ENTRY["feature-flagging-agent<br/>agent lifecycle"] --> RUNTIME["feature-flagging-agent-runtime<br/>Remote Configuration,<br/>EVP transport, trace adapter"]
        RUNTIME --> TELEMETRY["feature-flagging-telemetry<br/>exposure deduplication,<br/>span-tag state and encoding"]
    end

    RUNTIME <--> BRIDGE
Loading

Both delivery sources feed the same provider-owned evaluator:

flowchart TD
    CDN["Default: Datadog CDN"] --> HTTP["HTTP source"]
    RC["Explicit: Remote Configuration"] --> AGENT["Java agent"]
    AGENT --> BRIDGE["Raw byte[] bridge"]

    HTTP --> CORE["Core snapshot and evaluator"]
    BRIDGE --> CORE
    CORE --> RESULT["OpenFeature result"]
Loading
  • CDN delivery starts during provider initialization and stops during provider shutdown.
  • One reference-counted runtime is shared per application classloader.
  • Remote Configuration crosses the classloader boundary as raw UFC bytes.
  • Provider-owned UFC models never enter the bootstrap classloader.
  • The legacy typed bridge remains for released-provider compatibility.
  • A new provider reports a clear initialization error when an old agent lacks the raw bridge.
  • feature-flagging-lib is now feature-flagging-agent-runtime.
  • The JDK-only feature-flagging-telemetry module owns exposure deduplication and span-tag state and encoding.
  • Agent adapters and transports remain in feature-flagging-agent-runtime.

feature-flagging-telemetry is not embedded in dd-openfeature yet. The provider does not emit telemetry through it in this change.

Decisions

  • Keep parsing and evaluation in feature-flagging-core. Future CDN, Remote Configuration, offline, tracer API, and Lambda entry points can reuse one implementation.
  • Keep fetching in feature-flagging-http. Offline and Remote Configuration sources do not inherit HTTP dependencies or polling behavior.
  • Keep telemetry mechanics out of core. UFC evaluation remains independent from telemetry state and transport.
  • Extract agent-independent telemetry mechanics into feature-flagging-telemetry. The module depends only on the JDK.
  • Rename feature-flagging-lib to feature-flagging-agent-runtime. The old module was never a reusable library.
  • Keep Remote Configuration, EVP transport, trace integration, and agent lifecycle code in agent modules.
  • Publish no new Maven coordinate. Applications continue to depend only on dd-openfeature.
  • Keep environment and system-property parsing in the API adapter. The core receives explicit immutable options.
  • Keep the agent boundary narrow. Raw bytes and JDK values avoid shared model types and classloader linkage failures.
  • Preserve Remote Configuration instead of replacing it. New providers use the raw bridge. Released providers continue to use the legacy bridge with new agents.

This organization is the last required configuration and evaluator ownership refactor. It is not the final telemetry architecture.

The bootstrap bridge abstracts provider-to-agent communication across classloaders. It does not abstract local relay or direct intake transport. It also cannot create or enrich a trace when no tracer exists.

Next steps

The provider must own the agentless telemetry runtime because it is the only component present in provider-only deployments. The agent runtime remains an optional adapter.

Before agentless telemetry delivery, the follow-up should:

  • Embed feature-flagging-telemetry in dd-openfeature.
  • Capture one immutable evaluation event instead of using separate exposure, metric, and span hooks.
  • Use one bounded asynchronous pipeline for exposure deduplication, evaluation aggregation, and shutdown draining.
  • Share that pipeline through the existing application-classloader reference count.
  • Add versioned telemetry capabilities and an accepted/not-accepted result to the raw bridge.
  • Select each signal route independently. EVP availability does not prove OTLP or trace availability.
flowchart TB
    EVAL["Provider evaluation"] --> EVENT["Immutable evaluation event"]
    EVENT --> PIPE["Provider-owned telemetry runtime"]

    PIPE --> EVP["Exposure and aggregate EVP"]
    PIPE --> METRIC["Evaluation OTLP metric"]
    PIPE --> TRACE["Trace enrichment"]

    EVP --> EVP_ROUTE{"Agent accepts EVP?"}
    EVP_ROUTE -- Yes --> EVP_AGENT["Raw bridge to agent writer"]
    EVP_ROUTE -- No --> EVP_DIRECT["Local relay or direct EVP"]

    METRIC --> METRIC_ROUTE{"Local OTLP configured?"}
    METRIC_ROUTE -- Yes --> OTLP_LOCAL["OTLP HTTP on 4318"]
    METRIC_ROUTE -- No --> OTLP_DIRECT["Direct OTLP metrics intake"]

    TRACE --> TRACE_ROUTE{"Trace producer?"}
    TRACE_ROUTE -- "Datadog Java agent" --> TRACE_AGENT["Raw bridge to local-root span"]
    TRACE_ROUTE -- "OpenTelemetry SDK" --> TRACE_OTEL["Future OpenTelemetry span adapter"]
    TRACE_ROUTE -- None --> TRACE_NONE["No trace enrichment"]
Loading

The required follow-up depends on the deployment:

Deployment Supported by this change Required for future telemetry
Datadog Java agent attached CDN or Remote Configuration evaluation; existing agent span bridge remains available Add bridge capabilities and keep EVP, trace, and metric agent adapters
AWS Lambda with Datadog layers Provider-only evaluation can omit the Java layer; the normal tracing setup still attaches the Java agent Use the agent path when attached; use direct EVP and OTLP when it is absent
Cloud Run, Cloud Run Functions, or Azure Container Apps with serverless-init Provider-only evaluation does not require serverless-init or the Java agent Discover EVP separately; use OTLP on 4318 only when explicitly enabled
First-generation Cloud Run Functions or Azure Functions Provider-only evaluation can omit both documented Java agents Use direct EVP and OTLP; trace enrichment still needs a Java agent or OpenTelemetry trace SDK
Any JVM with no trace SDK Flag evaluation works Emit EVP and metrics, but do not claim trace enrichment because no trace exists

Direct OTLP trace intake makes an agentless trace path possible, but dd-openfeature must not become a tracer. A future OpenTelemetry adapter can enrich an active OpenTelemetry span. The application must still install a trace SDK and exporter.

PR risk

LLM-assisted static assessment: mixed change, high implementation surface, and moderate residual risk. This is not a move-only PR.

The customer-facing goal is narrow. CDN evaluation can run without the Java agent, while Remote Configuration remains available. The implementation is substantial because the previous design made the agent the configuration and lifecycle owner.

Assessment Result
Net-new customer behavior Yes
Mechanical code movement Yes, but it does not describe the full PR
Net-new telemetry delivery No
Implementation surface High
Residual risk after current validation Moderate

Net-new or changed behavior:

  • dd-openfeature can fetch, parse, store, and evaluate CDN configuration without -javaagent.
  • The provider now resolves the configuration source and owns source startup and shutdown.
  • Multiple providers in one application classloader share one reference-counted runtime.
  • Remote Configuration can send raw UFC bytes across a new classloader-safe bridge.
  • A new provider returns an explicit compatibility error when an attached agent lacks that bridge.
  • The provider JAR now embeds the internal core and HTTP implementations.

Moved or ported behavior:

  • UFC evaluation moved from DDEvaluator into feature-flagging-core. The expected OpenFeature results remain unchanged, but the implementation and model changed.
  • CDN polling moved from an agent-owned OkHttp implementation to a provider-owned Java 11 HttpClient implementation.
  • ETag, 304, retry, timeout, overlap prevention, scheduling, and shutdown behavior were ported rather than added for the first time.
  • Remote Configuration subscription, legacy typed parsing, EVP transport, and trace integration remain agent-owned.
  • The feature-flagging-lib rename and telemetry extraction are primarily structural. They do not add telemetry delivery.

The main residual risks are UFC semantic drift, provider lifecycle leaks, source-selection compatibility, HTTP behavior differences, and cross-classloader compatibility. The dogfooding proof, system tests, child-JVM tests, last-known-good tests, and compatibility matrix reduce these risks. They do not make the change risk-free.

This assessment comes from an LLM review of the static diff and recorded validation. It is not a formal safety proof or a substitute for human review.

Validation

  • ffe-dogfooding #102 validated Java commit 8bfa6f072a97ac8a7d4977c42e04fe8d63652e25.

    • Java 1.65.0-SNAPSHOT ran beside published Node.js dd-trace 5.118.0.
    • The Compose command started only the Java app, Node.js app, and evaluator. It did not start the Datadog Agent.
    • Java started without -javaagent and reported javaAgentAttached=false.
    • Java reached PROVIDER_READY in 1,094 ms. Node.js reached PROVIDER_READY in 706 ms.
    • The evaluator recorded 232 successful requests, zero failures, and a 100% success rate.
    • Java and Node.js returned true for the same configured flag.
    • Java provider shutdown completed and reported providerShutdown=true.
    • The local and Git-ref Docker builds installed the provider with its generated Maven POM.
  • system-tests #7300 used java@1.65.0-SNAPSHOT+8bfa6f072a.

    • All 30 configuration-source cases have pass evidence.
    • The parallel run passed 28 cases. Two workers failed during local test-agent TLS setup.
    • The affected cases and their parameterized companions passed on the targeted rerun: 4 passed.
    • All available Java FFE scenarios completed: 31 passed, 3 skipped, 1 deselected, 9 expected failures, and 3 expected passes.
    • The expected markers map to FFL-2446, FFL-1729, and FFL-2184.
    • The provider-only CDN scenario passed in 2.10 seconds.
    • The application reported javaAgentAttached=false.
    • The controlled CDN received the expected request.
    • The evaluated flag returned the configured on-value.
  • The compatibility matrix passed:

    • New provider + no agent + CDN.
    • New provider + released 1.64.2 agent + CDN.
    • New provider + new agent + Remote Configuration.
    • Released 1.64.2 provider + new agent + Remote Configuration through the legacy bridge.
    • New provider + released 1.64.2 agent + Remote Configuration returned the expected unsupported-bridge error.

@leoromanovsky leoromanovsky added type: refactoring comp: remote config Configuration at Runtime tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Jul 29, 2026
@datadog-datadog-prod-us1-2

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.01 s 13.91 s [-0.0%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent 12.86 s 12.98 s [-1.7%; -0.0%] (maybe better)
startup:petclinic:appsec:Agent 17.43 s 17.03 s [+1.3%; +3.3%] (significantly worse)
startup:petclinic:iast:Agent 17.30 s 17.49 s [-2.1%; -0.1%] (maybe better)
startup:petclinic:profiling:Agent 17.43 s 17.51 s [-1.7%; +0.8%] (no difference)
startup:petclinic:sca:Agent 17.43 s 17.25 s [-0.2%; +2.2%] (no difference)
startup:petclinic:tracing:Agent 16.48 s 16.66 s [-2.3%; +0.1%] (no difference)

Commit: 8bfa6f07 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature comp: remote config Configuration at Runtime tag: ai generated Largely based on code generated by an AI or LLM type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant